Learning Email Procedures for the Desktop

In this electronic age, we are all knowledge workers, tackling on daily basis the information that flows through our email, file systems, web browsers, calendars, and various other desktop applications. Email has come to be the center of desktop activity for many of us: we set up meetings, exchange documents, manage projects, forward interesting links, track requisitions, and perform a whole host of tasks through email. Many of these activities involve information flowing from one action to the next, across applications. They are also often repetitive, structured procedures, involving an ordered set of steps, many but not all of which are amenable to automation. In this paper, we present our approach to learning generalized dataflow procedures on the desktop from demonstrations by the user. Our system LAPDOG runs on the CALO desktop assistant and is capable of acquiring fully or partially automated procedures, spanning multiple applications, from demonstrations where the user may perform some unobservable actions.